fix(crates-sync): reap git zombie processes and harden worker loop - #2176
Conversation
The import Job died after ~2h with 85k "push rejected" failures that never reached mono-engine. Root cause: Python runs as PID 1 and never reaps adopted orphans — every git HTTP op leaves a `git remote-http` shim behind, so zombies accumulate (~36/s at full speed) until the pod hits its pids cgroup limit (19122) and every fork fails with EAGAIN. git push then exits non-zero before sending any HTTP, and an uncaught BlockingIOError from subprocess.run kills a worker thread, letting the run "finish" at 5.7% with the rest of the queue stranded. - Dockerfile: run tini -g as PID 1 to reap orphaned git helpers - _run_cmd: retry subprocess spawn on OSError (EAGAIN/ENOMEM) so one fork failure no longer kills a worker thread - worker_loop: per-item catch-all turns a bad item into a "fail" record instead of a dead thread stranding the queue - fail-fast breaker: 500 consecutive failures abort the run with exit code 3 instead of burning t queue into false "fail" records - git_push_main: always log a bounded stderr detail on rejection (was VERBOSE-only, which hid the root cause)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12b703422b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| saved_replicas = scale_deployments( | ||
| scale_names, to_zero=True, saved=None, dry_run=args.dry_run | ||
| ) |
There was a problem hiding this comment.
If scaling fails after an earlier deployment was already set to zero—for example, mono-engine scales down successfully but the default orion-server deployment is absent—scale_deployments raises before returning, so this assignment never updates saved_replicas. The exception handler then skips restoration because the dictionary is still empty, leaving mono-engine scaled to zero indefinitely; persist each deployment's replica count before performing the scale operation so partial failures can be rolled back.
Useful? React with 👍 / 👎.
| if rm.returncode not in (0,): | ||
| # Empty prefix may still exit 0; treat non-zero as warning if "does not exist" | ||
| err = (rm.stderr or "") + (rm.stdout or "") | ||
| if "does not exist" in err.lower() or "not found" in err.lower(): | ||
| print(f"Prefix {prefix}/ absent; ok") | ||
| else: | ||
| print(f"WARN: mc rm {prefix}/ exited {rm.returncode}") |
There was a problem hiding this comment.
When mc rm returns nonzero because of an authorization, network, or RustFS failure, this branch only prints a warning; execution subsequently prints S3_WIPE_OK, restarts the writers, and reports the reset as successful. Because PostgreSQL has already been truncated at this point, the cluster can be brought back with stale Git/LFS objects and an incomplete destructive reset; non-absence errors should raise and prevent successful completion.
Useful? React with 👍 / 👎.
2 participants
No description provided.